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\0 EXPERIENTIAL INSTRUCTION OF LISTENING FOR COLLEGE STUDENTS: 
^ THEORETICAL AND PRACTICAL APPROACHES TO TEACHING LISTENING 

Q 

a 

Evaluating Listening Texts 



Paul Kaufmann 
Iowa State University 



ABSTRACT 



Eight college listening texts were selected and evaluated by 
the following criteria: Organization, Supporting Material, 
and Readability. Text organization was assessed by preface 
material, table of contents, index, and references. Text 
supporting material was evaluated by the use of hard data, 
soft data, visual aids, and bibliographies. Text 
readability was judged by transitions, tone, visual 
structure, and reading level. Worksheets were developed for 
each of these criteria, and numerical scores were assigned 
to each of the texts being evaluated. 
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OVERVIEW 



This paper explains a process for evaluating college 
listening texts. Since those interested in teaching and 
researching listening might benefit from this process, the 
steps are detailed in this two part report. The first 
section discusses the pre-evaluation steps (selecting texts 
and developing evaluation criteria); the second section 
reports the evaluation results. 



PRE-EVALUATION STEPS 



Two steps preceded the actual text evaluation. First was 
the selection of the texts to be evaluated, and second, the 
development of evaluation criteria. 



TEXT SELECTION 



In choosing the listening texts for evaluation, three 
criteria were considered: 1) the text was appropriate for a 
college course, that is, it contained chapters, headings, a 
preface, and focused on the topic of listening as applied to 
understanding, personal interactions, teaching and learning, 
management, and negotiation. Highly specialized listening 
texts — such as listening to music, birds, or whales — were 
not selected for evaluation. Two, the text was 
current — published within the past decade. Three, texts had 
100 pages, minimum. Based on these criteria, eight texts 
were selected: 



Text 1: Burley-Allen, Madelyn, Listening: the Forgotten 
Skill , New York: John Wiley & Sons, Inc. 1982, 

Text 2: Bostrom, Robert N, Listening Behavior: Measurement 

and Application , New York: The Guilford Press, 1990, 

Text 3: Borisoff, Deborah and Michael Purdy. Listening in 

Everyday life: a Personal and Professional Approach , 
London: University Press of America, 1991, 

Text 4: De Marc, George, Lyman K, Steil and Joanne Summerfield. 
Listening: it can Change your Life: a Handbook for 
Scientists and Engineers . New York: Wiley & Co,, 
1983 

Text 5: Maidment, Robert. Tuning in: a Guide to Effective 

L istening . Gretna, LA: Pelican Publishing Company, 
1984. 

Text 6: Wolff, Florence I. and Nadine C. Marsnick. Perceptive 
Listening . New York: Holt, Rinehart and Winston, 
Inc., 1992. 
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Text 7: Wolvin, Andrew and Carolyn Gwynn Coakley. Listening , 
4th ed. Dubuque, Iowa: William C. Brown, 1992. 

Text 8: Kaufmann, Paul J. Sensible Listening: the key to 
Responsive Interaction , 2d ed. Dubuque, Iowa: 
Kendall Hunt Publishing Co., 199 3. 

CRITERIA FOR EVALUATION 

Central to the task of evaluating these listening texts was the 
development of appropriate criteria. Here, were a myriad of 
possibilities. In wading through the alternatives, one needed to 
consider criteria that could be consistently and reasonably 
applied in a timely manner. 

An initial thought was to set-up a criterion called, "Thorough 
and Accurate Presentation of Listening Theory," but this was not 
workable because few generally accepted theories mark the study 
of listening. Unlike a text about descriptive statistics where 
one would expect to find mean, median, mode, and range discussed, 
one text about listening might cover gender differences and 
recall, while another might cover comprehensive and therapeutic 
listening, but not gender and recall. Consequently, an evaluation 
criterion directly targeting listening theories was not used in 
this text evaluation. 

Three main evaluative criteria with four subcategories for each 
were developed. The first main criteria is Organization , with 
the four subcategories A) Preface, B) Table of Contents, C) 
Index, and D) References. Supporting Material is the 
second main category, with subcategories A) Hard Data, B) Soft 
Data, C) Visual Aids, and D) Bibliography. The third main 
category is Readability , with subcategories A) Transitions, B) 
Tone, C) Visual Structure, and D) Reading Level. 

For consistency of evaluation and inter-rater reliability, an 
evaluation worksheet was developed. Tile worksheet explained each 
of the three evaluation criteria and provided the evaluators a 
form to record their evaluations. The criteria explanations and 
example worksheets follow. 

Organization. --This category gives direction to the text user by 
dividing the text into manageable sections of information. The 
four subcategories of organization are 1) preface, 2) table of 
contents , 3 ) index , and 4 ) references . 
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The preface is made-up of the introductory or forwarding 
comments, and sometimes a personax note to the reader. The 
preface may include historical background, explanation of 
overarching concepts and suggestions for approaching the covered 
material. The table of contents outlines the significant 
topics and provides the reader with a schema to map the 
information. 

The index allows the reader to reference key terminology, 
concepts, and names by giving their location in the text. 
Similarly, the reference section may contain a glossary, 
appendixes, and notes. 

Supporting Material . --Through 1) hard data, 2) soft data, 3) 
visual aids, and 4) bibliographies, supporting material lends 
clarity and understandability to the key ideas in the 
text. Hard data , the first subcategory of supporting material, 
are experimental and statistical. They give support through 
measurable facts. Soft data also give support through 
testimony, quotes, anecdotes, and illustrations. The third 
subcategory, visual aids , demonstrates hard and soft data 
information through tables, graphs, cartoons and pictures. 
Lastly, bibliographies are listings of the external sources for 
the text, and allow the reader to research and verify the 
information. 

Readability. — Readability is the component of the text that 
establishes and maintains the author's relationship with the 
intended reader. A first means of accomplishing readability is 
through transitions which provide forecasts, summations, and 
topic changes. Transitions give continuity to the text. 
The second subcategory of readability is tone , which 
includes the use of active, vivid language and the degree of 
formality the author employs. Third is visual structure which 
involves the appeal of the print (font/size), column width, 
paragraph length, and page layout to the reader. Visual structure 
should not be confused with visual aids as defined under 
supporting material. Subcategory four is reading level which 
considers the appropriateness of the vocabulary for the readiness 
level of the intended reader. 

With the three criteria and their four subcategories defined, the 
next step was to prepare worksheets for the text evaluations. 
Considering the tendency toward midrange responses with Likert 
scales, the worksheet was a means to provide the evaluator 
assurance that extreme values could be justified. An odd 
numbered scale of one (1) being poor, to five (5) being better, 
was chosen for each of the three general evaluation categories — 
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ORGANIZATION, SUPPORTING MATERIAL, AND READABILITY, and a two 
part checklist was designed for each of the four subcategories 
under each of these . (See worksheets, p. 6.) 

To help assure inter-rater reliability, if an evaluator found 
that a text had none of the two items under a subcategory, a zero 
(0) or one (1) was recorded. If a text met one of the two items, 
a rating of two (2) or three (3) was entered, and if it met both 
of the subcategory items, a rating of four (4) or five (5) was 
entered. Written comments were requested to explain scores 
within these ranges. 

This evaluation system accomplished two purposes: first, it 
completed the goal of having some degree of objectivity by 
demonstrating the range in which a score would lie. Second, it 
incorporated subjectivity by allowing the evaluator to correlate 
the even numbered checklist with the odd numbered Likert scores. 
This is realistic because some unique criteria may not be 
expressed in the checklist but may indicate a higher or lower 
degree of quality. These differences in text quality, then, can 
be expressed in the higher or lower choice of Likert values. The 
comments gave insightful reasons to these unique evaluative 
factors and encouraged the evaluator to use the end scores of the 
Likert scale comfortably. 

Before closing discussion of evaluation criteria, it is important 
to touch on two topics that were excluded as evaluation 
categories: Clarity and Listening Theory. Certainly both of 
these concepts are important in listening textbooks. Fortunately, 
their impact can be indirectly judged by the three main 
evaluation categories. For example, clarity is created by 
appropriately organizing concepts, effectively supporting them, 
and insuring the~readability of the presentation. Further, 
theories that are valid and durable are likely to be well 
organized, and, more importantly, supported with fact, data, and 
experimentation while presented at an appropriate readability 
level . 

After the criteria were established, condensed evaluation report 
forms were prepared for each category — Organization, Supporting 
Material, and Readability: 
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EVALUATION REPORT FORM: ORGANIZATION- -OVERALL RATING: 

Preface : 

Author gives historical background and explanation of the 

thought process for developing text and gives validation 
for the course of study. 

Includes introductory or forwarding comments and personal 

notes or suggestions to readers. 
COMMENTS : 



Table of Contents: 

Outlines significant topics in form of headings and 

subheadings . 

Gives a chapter introduction which provides the reader 

with a schema to map the information. 
COMMENTS : 



Index: 

Allows reader to reference key terminology and key 

concepts and their location in the text 

Additionally, index provides opportunity to look up 

key individuals (author index) 
COMMENTS : 



References : 

A glossary is given that provides definition relative 

to the text. 

Additional appendixes endnotes, footnotes, or chapter 

notes are provided to give further references. 
COMMENTS : 



OVERALL COMMENTS—ORGANIZATION : 
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Rater/Reviewer: 

EVALUATION REPORT FORM: SUPPORTING MATERIAL—OVERALL RATING: 

Hard Data: 

Experiments are descriptive and sufficient in design and 

conclusion. 

Statistical data gives support through measurable facts. 



COMMENTS : 



Soft Data: 

Testimony and quotes are provided that support the 

material . 

Anecdotes and verbal illustrations give real-life 

examples 

to help clarify material. 
COMMENTS : 



Visual Aids: 

Tables and graphs help to digest the hard data into 

understandable formats. 

Cartoons and pictures are provided for additional support 

and visual testimony. 
COMMENTS : 



Bibliography : 

Variety of sources i.e. journals, texts, and interviews 

has been included in research to provide complete, 
well-rounded foundation 

While noting the value of early theorists the 

concentration of the text is on recent material. 
COMMENTS : . 



OVERALL COMMENTS - - S U P PORTI NG MATERIAL: 



ERIC 



8 



4 



ERIC 



Listening texts, p. 7 



Rater/Reviewer : 



EVALUATION REPORT FORM: READABILITY- -OVERALL RATING: 

Transitions: 

Key Ideas are forecast and summarized. 



Smooth topic changes lend to continuity, 



COMMENTS : 



_Tone : 

Text uses action verbs as well as vivid and succinct 

descriptions . 



COMMENTS 



Expressed intent by establishing familiarity through use 
"of formal or informal language. 



_Visual Structure : 

The size and font of print are appropriate to level of 

text and comfortable to the reader. 



COMMENTS 



Page layout (column width, paragraph length) is 
"conductive to reading. 



_Reading Level: 

Key terms are identified by bold print or italics. 



COMMENTS ; 



Context lends understanding to ambiguous terms 
"and/or words are explicitly defined within the text, 



OVERALL COMMENTS — READABILITY: 
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THE EVALUATION OF LISTENING TEXTS 



Once criteria were developed and formatted, the texts were 
evaluated. Sixteen students in a graduate listening seminar 
evaluated the texts. Each text was evaluated by two students, and 
the two scores were averaged for each category. The data were 
graphed for text comparison: 
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DATA SUMMARY 

Book 1: Burley-Allen. Listening: the Forgotten Skill 



I . ORGANIZATION 

A. Preface 

B. Table of Contents 

C. Index 

D. References 

II . SUPPORTING MATERIAL 

A. Hard Data 

B. Soft Data 

C. Visual Aids 

D. Bibliography 



3.4 



4.0 
3.5 
2.5 
3.5 



4.5 
5.0 
5.0 
4.0 



4.6 



III. READABILITY 

A. Transitions 

B. Tone 

C. Visual Structure 

D. Reading Level 



4.8 



4.0 
5.0 
5.0 
5.0 

DATA SUMMARY 



Book 2: Bostrom. Listening Behavior: Measurement and 

Application. 



ORGANIZATION 

A. Preface 

B. Table of Contents 

C. Index 

D. References 



4.5 
4.5 
4.5 
4.0 



4.4 



II. SUPPORTING MATERIAL 

A. Hard Data 

B. Soft Data 

C. Visual Aids 

D. Bibliography 

III. READABILITY 

A. Transitions 

B. Tone 

C. Visual Structure 

D. Reading Level 



4.8 



5 , 
5 
5 
5 



0 
0 

.0 
. 0 



5.0 
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DATA SUMMARY 

Book 3: Borisoff and Purdy. Listening in Everyday life: 
a Personal and Professional Approach , 

I. ORGANIZATION 3.5 

A. Preface 4,0 

B. Table of Contents 5.0 

C. index 2.0 

D. References 3.0 

II. SUPPORTING MATERIAL 3.8 

A. Hard Data 4.0 

B. Soft Data 4.0 

C. Visual Aids 3.0 

D. Bibliography 4.0 

III. READABILITY 5.0 

A. Transitions 5.0 

B. Tone 5.0 

C. Visual Structure 5.0 

D. Reading Lsvel 5.0 

DATA SUMMARY 

Book 4: DeMarc, Steil, and Summer field. Listening: it 

can Change your life: a Handbook for Scientists 
and Engineers . 

I. ORGANIZATION 4.1 

A. Preface 4 . 0 

B. Table of Contents 4.E 

C. Index 4.5 

D. References 3.5 

II . SUPPORTING MATERIAL 3.9 

A. Hard Data 2.0 

B. Soft Data 4.5 

C. Visual Aids 5.0 

D. Bibliography 4.0 

III. READABILITY 4.5 
A. Transitions 4.5 

B . Tone 4 . 5 

C. Visual Structure 4.5 

D. Reading Level 4.5 
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DATA SUMMARY 

Book 5: Maidment. Tuning in: a Guide to Effective Listening < 

I. ORGANIZATION 2.3 
A. Preface 4 . 0 

B* Table of Contents 3.0 

C. Index 1*0 

D. References 1.0 

II. SUPPORTING MATERIAL 3.5 

A. Hard Data 4.0 

B. Soft Data 5.0 

C. Visual Aids 4.0 

D. Bibliography 1.0 

III. READABILITY 3.3 

A. Transitions 1.0 

B . Tone 4 . 0 

C. Visual Structure 5.0 

D. Reading Level 3.0 

DATA SUMMARY 

Book 6: Wolff and Marsnick. Perceptive Listening . 

I. ORGANIZATION 3.9 

A. Preface 3 . 5 

B. Table of Contents 5.0 

C. Index 5.0 

D. References 2 . 3 

II. SUPPORTING MATERIAL 4.6 

A. Hard Data 4.0 

B. Soft Data 5.0 

C. Visual Aids 5.0 

D. Bibliography 4.3 

III. READABILITY 4.8 

A. Transitions 5 . 0 

B. Tone 5.0 

C. Visual Structure 4.3 

D. Reading Level 5.0 
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DATA SUMMARY 
Book 7: Wolvin and Coakley. Listening . 4th ed, 



ORGANIZATION 

A. Preface 

B. Table of Contents 

C. Index 

D. References 



5.0 
5.0 
5.0 
4.0 



4.8 



II . SUPPORTING MATERIAL 

A. Hard Data 

B. Soft Data 

C. Visual Aids 

D. Bibliography 

III. READABILITY 

A. Transitions 

B. Tone 

C. Visual Structure 

D. Reading Level 



4.5 



4.0 
5.0 
4.0 
5.0 



5.0 
4.0 
3.5 
4.0 

DATA SUMMARY 



4.1 



Book 8: Kaufmann. 



Sensible Listening: the key to Responsive 
Interaction. 



ORGANIZATION 

A. Preface 

B. Table of Contents 

C. Index 

D. References 



5.0 
4.0 
4.0 
3.0 



4.0 



II. SUPPORTING MATERIAL 

A. Hard Data 

B. Soft Data 

C. Visual Aids 

D. Bibliography 

III. READABILI TY 

A. Transitions 

B. Tone 

C. Visual Structure 

D. Reading Level 



4.8 



0 

.0 
.0 
,0 



0 
0 
0 
0 



4.5 
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CONCLUSION 

Averaging the three main criteria scores, the texts are ranked in 
the following order: 

#1 Bostrom. Listening Behavior: Measurement and Application . 
4.73 of a possible score of 5 

#2 Wolvin and Coakley. Listening . 

4.46 of a possible score of 5 

#3 Wolff and Marsnick. Perceptive Listening . 

4.43 of a possible score of 5 

tie 

#3 Kaufmann. Sensible Listening: the key to Responsive 
Interaction . 

4.43 of a possible score of 5 

#4 Burley-Allen. Listening: the Forgotten Skill . 

4.23 of a possible score of 5 

#5 DeMarc, Steil, and Summerfield. Listening: it 
can Change Your life . 
4.16 of a possible score of 5 

#6 Borisoff and Purdy. Listening in Everyday life: a 

Personal and Professional Approach 
4.10 of a possible score of 5 

#7 Maidment. Tuning in: a Guide to Effective Listening . 
3.03 of a possible score of 5 

Evaluating listening texts is a individual process, where 
instructors select a text to meet the specific needs and 
backgrounds of their students. The scheme presented in this 
paper for evaluating texts can be a helpful first step in making 
that final text selection. 
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